Papers with Automated evaluation
FineSurE: Fine-grained Summarization Evaluation using LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text summarization evaluation do not correlate well with human judgments . evaluators that use Likert scale scores are limited in their ability to perform deeper analysis. |
| Approach: | They propose a fine-grained evaluator specifically tailored for the summarization task using large language models. |
| Outcome: | The proposed method improves on open-source and proprietary LLMs and shows better completeness and conciseness than existing methods. |
InstaJudge: Aligning Judgment Bias of LLM-as-Judge with Humans in Industry Applications (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Automated evaluation using LLM-as-Judge is a viable alternative to human evaluation, but misalignment of judgment biases between humans and LLMs hinders its use in real-world applications. |
| Approach: | They propose an LLM-as-Judge library that improves alignments of judgment biases through automatic prompt optimization. |
| Outcome: | The proposed library outperforms existing LLM-as-Judge libraries by a large margin while being more cost efficient. |
Adaptive-VP: A Framework for LLM-Based Virtual Patients that Adapts to Trainees’ Dialogue to Facilitate Nurse Communication Training (2025.findings-acl)
Copied to clipboard
Keyeun Lee, Seolhee Lee, Esther Hehsun Kim, Yena Ko, Jinsu Eun, Dahee Kim, Hyewon Cho, Haiyi Zhu, Robert E. Kraut, Eunyoung E. Suh, Eun-mee Kim, Hajin Lim
| Challenge: | standardized patient (SP) simulations are costly and inflexible. |
| Approach: | They propose a framework that leverages large language models to dynamically adapt VP behavior based on trainee input. |
| Outcome: | The proposed framework reflects real-world communication skills and produces more natural and realistic interactions than existing methods. |
From Shijing to English and German: Resources and Evaluation for LLM Translation of Early Chinese Poetry (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) show promise in literary translation, but their performance in poetry remains unexplored. |
| Approach: | They propose a framework that integrates knowledge-driven, rule-based, and LLM-as-judge metrics into a Shijing corpus . their code, lexical KB, and corpus reconstruction protocols are available at https://github.com/ML-KULeuven/ShijingLLMTrans. |
| Outcome: | The proposed framework achieves higher human correlation than traditional metrics and high statistical stability. |
CourtEval: A Courtroom-Based Multi-Agent Evaluation Framework (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing automated evaluation metrics like ROUGE and BLEU show low correlation with human judgments. |
| Approach: | They propose a multi-agent evaluation framework that integrates multiple agents . they use ROUGE and BLEU to evaluate natural language models . |
| Outcome: | The proposed evaluation framework outperforms the current state-of-the-art methods in two meta-evaluation benchmarks. |